Determining Critical Topics for Undergraduate Surgical Education in Rwanda: Results of a Modified Delphi Process and a Consensus Conference

Background Developing a contextually appropriate curriculum is critical to train physicians who can address surgical challenges in sub-Saharan Africa. An innovative modified Delphi process was used to identify contextually optimized curricular content to meet sub-Saharan Africa and Rwanda’s surgical needs. Methods Participants were surgeons from East, Central, Southern, and West Africa and general practitioners with surgical experience. Delphi participants excluded or prioritized surgical topic areas generated from extensive grey and formal literature review. Surgical educators first screened and condensed identified topics. Round 1 screened and prioritized identified topics, with a 75% consensus cut-off based on the content validity index and a prioritization score. Topics that reached consensus were screened again in round 2 and re-prioritized, following controlled feedback. Frequencies for aggregate prioritization scores, experts in agreement, item-level content validity index, universal agreement and scale-level content validity index based on the average method (S-CVI/Ave) using proportion relevance, and intra-class correlation (ICC) (based on a mean-rating, consistency, two-way mixed-effects model) were performed. We also used arithmetic mean values and modal frequency. Cronbach's Alpha was also calculated to ascertain reliability. Results were validated through a multi-institution consensus conference attended by Rwanda-based surgical specialists, general practitioners, medical students, surgical educators, and surgical association representatives using an inclusive, participatory, collaborative, agreement-seeking, and cooperative, a priori consensus decision-making model. Results Two-hundred and sixty-seven broad surgical content areas were identified through the initial round and presented to experts. In round 2, a total of 247 (92%) content areas reached 75% consensus among 31 experts. Topics that did not achieve consensus consisted broadly of small intestinal malignancies, rare hepatobiliary pathologies, and transplantation. In the final round, 99.6% of content areas reached 75% consensus among 31 experts. The highest prioritization was on wound healing, fluid and electrolyte management, and appendicitis, followed by metabolic response, infection, preoperative preparation, antibiotics, small bowel obstruction and perforation, breast infection, acute urinary retention, testicular torsion, hemorrhoids, and surgical ethics. Overall, the consistency and average agreement between panel experts was strong. ICC was 0.856 (95% CI: 0.83-0.87). Cronbach's Alpha for round 2 was very strong (0.985, 95% CI: 0.976-0.991) and higher than round 1, demonstrating strong reliability. All 246 topics from round 4 were verbally accepted by 40 participants in open forum discussions during the consensus conference. Conclusions A modified Delphi process and consensus were able to identify essential topics to be included within a highly contextualized, locally driven surgical clerkship curriculum delivered in rural Rwanda. Other contexts can use similar processes to develop relevant curricula.


Introduction
Low-and middle-income countries (LMICs) experience a deficit of approximately 143 million surgical procedures each year, and surgical provider shortfalls contribute significantly to this huge gap in access to care [1]. These deficits are global but are more pronounced in sub-Saharan Africa (SSA) [2,3]. While the recommended surgeon, anesthetist, and obstetrician (SAO) density is 20 per 100,000 population, SSA has less than two surgical specialists per 100,000. The challenge is even more pronounced in Eastern, Central, and Southern Africa which has only 0.53 surgeons per 100,000 population [2,3]. Training of surgical providers needs to be prioritized in attempting to meet such SAO provider density targets.
Between 2016 and 2019, 24 Rwandan surgeons were certified by the College of Surgeons of Eastern, Central, and Southern Africa (COSECSA) [4]. In addition, several specialist trainees were certified through the University of Rwanda [5] and not-for-profit training programs like the Pan African Association of Christian Surgeons [6]. However, current surgical specialist volumes in Rwanda are still insufficient to handle the local surgical burden of disease of about 12,000 surgical conditions per 100,000 people, which is about 27,160 surgeries required per qualified surgeon in the country [7,8]. The role of non-specialist physicians in providing basic, life-saving surgical care in SSA is already established [9][10][11]. In Rwanda, general practitioners (in this context, medical doctors with no specialist residency training) in rural district hospitals perform most of the basic general surgery procedures, cesarean sections, and closed fracture manipulation [12]. As such, expectations and training needs for non-specialist doctors in LMICs differ from those in high-income countries (HICs) as there is a need to prepare medical graduates entering into general practice in resource-constrained locations for essential surgical procedures [9,10,13]. However, it has been observed that LMIC surgical training curricula are often simply adopted from HIC institutions with little alteration [14].
The University of Global Health Equity (UGHE) is a new health sciences university based in rural Rwanda [15]. The overall vision for surgical training at UGHE is to equip medical students with the necessary skills to carry out Bellwether surgical procedures (cesarean section, laparotomy, and management of open fractures) within the scope of their practice [16], manage surgical emergencies, and appropriately refer surgical patients to higher levels of care when required. The University of Rwanda is the key player in the training of medical practitioners and specialists required to meet the country's surgical needs. It is the largest public training and research institution in Rwanda [17]. Both institutions collaborated to define and prioritize topics for undergraduate surgical training for their context. We describe a modified Delphi consensus process used to derive topical core surgical content areas to be used in curriculum development for undergraduate medical students in Rwanda.
The abstract of preliminary results of this work was submitted to the American College of Surgeons Scientific Conference on October 2022, presented at the 2022 Rwanda Surgical Society Annual Conference on November 19, 2022, and presented at the 2022 College of Surgeons of East, Central, and Southern Africa Scientific Conference on December 8, 2022.
Aims of the study were to derive topical core curriculum content areas to be used in curriculum development for undergraduate surgery trainees in SSA and Rwanda specifically, to identify priorities that will be relevant to local surgical practice and for preparing trainees for relevant rural surgical practice, and to generate consensus on undergraduate surgery core curriculum content for Rwanda.

Materials And Methods
With limited precedent regarding the development and delivery of contextualized undergraduate surgical curricula across SSA, there was a need to generate expert consensus on the topic. The Delphi technique is a well-established approach with three key characteristics that can be used to answer a research question, based on the consensus of subject matter experts [18]. Firstly, it is based on a series of rounds. Questions in each subsequent round are based on findings from the previous round, and the study evolves in response to earlier findings. Next, respondents can see the results of the previous rounds in order to permit them to reflect on others' views and possibly reconsider their own. Finally, results of each round are shared anonymously to avoid bias [18]. A modified Delphi process was used to develop expert consensus as there is consistent evidence to support the superiority of group decision-making over individual opinion when seeking expert judgment [19]. Of methods utilized to identify consensus and solicit group opinion, we selected a modified Delphi technique over nominal techniques of a simple survey or a strict consensus conference [20]. This technique permits the views of experts to be sought and combined without them necessarily having to meet; this was preferable considering the constraints of distance, time, and COVID restrictions.
We modified the Delphi technique in two ways: by developing a questionnaire for circulation to panelists based on a list obtained through a rigorous review of the literature (confirmatory approach), as opposed to the generation of the questionnaire by an expert Delphi panel, and by permitting new Rwanda-based panelists to join in the second round. We also combined inclusion/exclusion and prioritization (traditionally separate rounds) in each consolidated round. This confirmatory approach to the first round has been utilized in several studies through which content for undergraduate curricula in other areas has been obtained [21][22][23]. This study was guided by the Guidance on Conducting and REporting DElphi Studies (CREDES) recommendations [24].
Following the generation of the topic list through a literature review, two consolidated rounds of consensus surveying were undertaken followed by an in-person consensus conference. The course of this multi-step consensus study is demonstrated in Figure 1.

FIGURE 3: Geographic Distribution of Expert Panelists
For the second round, 87% (27) of the respondents worked in Rwanda. Eleven (35%) respondents were retained from the first round, and the male-to-female ratio was 4:1. Representation of expert general practitioners increased from 13% to 35%.

Modified Delphi round 1
Round 1 was undertaken with the aim of inclusion, exclusion, and prioritization of curriculum topic areas based on applicability to SSA general practice, unique purpose of the training institution, and feasibility in a low-income setting. The content validity index (CVI) was calculated to establish Delphi consensus from the ratings of panel members [27,28]. The CVI describes the percentage of respondents who rated the thematic areas as important or somewhat important. For topics that were included by a respondent, priority ranking (based on whether the topics were not important, somewhat important, or important) was carried out using a calculated aggregate priority score. Prioritization was performed by calculating an average score (in percentage) out of a total of 4 points (exclude = 1, not important = 2, somewhat important = 3, important = 4). Aggregate group prioritization to the level of "somewhat important" was set at a score of 3 out of 4 (75%). A prioritization ranking of 75% was set as the predefined consensus level and was used to determine which topics advanced through to the next round [29]. If a CVI >75% or aggregate prioritization score of >75% was attained, consensus was assumed. Finally, respondents were given the opportunity to suggest additional subject areas, not already included in the survey, that they felt should be incorporated into an undergraduate curriculum. Prioritized topics were advanced to the next round. Consistency and average agreement between panel experts were calculated. Intra-class correlation (ICC) was calculated using Shrout and Fleiss 1979 Convention 3,k consistency, and a two-way mixed-effects model. Cronbach's Alpha was also calculated to ascertain reliability.

Modified Delphi round 2
Consensus items were advanced to the next modified Delphi round from March to August 2022 to establish consensus between 31 respondents. Participants were given anonymized feedback on the topics that did and did not achieve consensus from the first round and were permitted to comment on these. Inclusion, exclusion, and re-prioritization were repeated for prioritized topics from the first round. A CVI of at least 75% was set as the predefined consensus level and used to determine which topics advanced through to the consensus conference. If this level was attained, consensus for this round was assumed, and further validation was not deemed necessary. Free text responses were encouraged to add unique perspectives to the round. Results from the second round were advanced to the Consensus conference. ICC and Cronbach's Alpha were also calculated.

Institutional review board approval and participant consent
The Modified Delphi consensus process was approved by the UGHE ethical review committee (UGHE-IRB/2021/059). Informed consent was obtained from all participants at every level of the consensus process. Electronic informed consent was obtained during the Delphi rounds, and written informed consent witnessed by the consensus conference secretariat was obtained from participants at the consensus conference. The participants were informed in writing about the rationale, methods, and aim of the Delphi and consensus conference and had the opportunity to ask clarifying questions at the venue.

Modified Delphi round 2 results
In the second round, 99.6% of content areas reached 75% consensus among 31 experts (Appendix 5). The highest prioritization was given to basic principles like the management of wounds and wound healing, fluids and electrolytes in surgery, physiological response to trauma, sepsis and infection, preoperative preparation, antibiotics use in surgery, ethics and surgery, surgical quality and safety, immediate postoperative care, and imaging in surgery. Abdominal conditions including appendicitis, typhoid enteritis, typhoid ileal perforation, small intestinal obstruction, gastric outlet obstruction, peptic ulcer diseases, and upper gastrointestinal bleeding were also high-priority areas. Pediatric conditions like intussusception, approach to bilious vomiting in the newborn, and pediatric fluid and electrolyte management were highly prioritized. Highest ranking anorectal conditions included hemorrhoids and anal fissures. Breast pathologies including cancer and infections, urologic conditions including acute urinary retention and hematuria, and orthopedic principles of fracture management were also ranked high priority for medical students in Rwanda (Appendix 5). Highest and lowest priority surgical topics are shown in Table 2  Scale-level content validity index based on the average method (S-CVI/Ave) using proportion relevance) = 0.935. Any S-CVI/Ave ≥ 0.9 implies excellent content validity. *Excluded.

Consensus conference
All 246 topics from the final round were presented, discussed, and verbally accepted by all 40 participants in open forum discussions during the consensus conference. Prioritization was accepted with no "stand asides" or blocking of consensus. The conference also came to a consensus to repeat the process every five years to maintain current relevance.

Discussion
Using this multistage consensus process, we primarily present a list of prioritized surgery education topics that are relevant to the Rwandan context. These content areas have been adopted by medical schools in Rwanda. These will help focus curriculum development and form the basis for discussions on both Entrustable Professional Activities (EPAs) for new medical schools and curriculum reform for established institutions [31]. EPAs are an emerging pedagogical tool gaining traction in the teaching and training of medical students and post-graduates globally to ensure competence and trustworthiness in discrete professional tasks [32]. Secondarily, based specifically on the results of the first round, we present a starting point for similar discussions for other SSA countries. While every context is inherently different, our initial literature review and the multi-country first round of consensus building can be adapted for discussion in other countries in SSA. This can serve as a springboard from which to begin answering questions about what should be taught in other parts of East Africa, and other African sub-regions.
The argument has not yet been made for a unified undergraduate curriculum across all of Africa. Unlike some recognized close-knit regions with similar health systems and medical qualifying examinations [24], SSA has a diversity of surgical pathology specific to geographical, cultural, and social contexts. For this reason, one unified curriculum for all of SSA may be inappropriate. However, the argument should be made for contextual undergraduate surgical curricula in various parts of SSA. Surgical training curricula for the Global South cannot simply be adopted with little alteration from institutions from the Global North. Various factors, including country-specific practice expectations, surgeon, anesthetist, and obstetrician density and distribution, burden and variety of surgical disease, the extent of pathology at presentation, resources for diagnosis and treatment, and the stage of training at which graduates begin direct care of surgical patients, vary [33,34]. Therefore, surgical educators should strive to contextualize and prioritize locally relevant epistemology, medical illustrations, textbooks, manikins, and simulation materials bearing in mind indigenous knowledge, local research, and relatable skin tones.
The most highly prioritized content from this study included universal principles and foundations of general surgical care, management of acute emergencies, and a wide range of common surgical pathologies. Hepatobiliary surgical pathologies and training on the management of complicated surgical patients (surgery in the cirrhotic patient) were largely perceived to be out of scope for the non-specialist general practitioner. Rarer pathologies in the specific context like obesity, colon carcinoid, pilonidal disease, and metabolic bone diseases were also de-emphasized. Indeed, the mantra that "common things occur commonly" is reflected in the consensus. Of note is the fact that although an introduction to laparoscopic surgery attained consensus, it was in the lowest fifth percentile for this curriculum focused on junior medical student clerks.
We also illustrate the role of non-specialist physician surgical providers in helping define surgical content [35]. General practitioners who had lived experience in rural district hospitals were involved in each round and the consensus conference. One recent national study shows that surgeons will often underestimate the importance of surgical topics and training for non-surgeons [36]. Thus, the inclusion of General Practitioners' voices grounded the study in the needs of their daily practice. Other studies in developing or reviewing medical school curricula have emphasized the need for general practitioners' input [35,37,38], as this is often the first phase of trainees' practice upon graduation.

Limitations
As part of our modification, more Rwandan practitioners and general practitioners were included in the second round and the consensus conference by design, and therefore, over 50% of participants were different from those involved in the first round. This has limited our ability to assess for internal consistency between rounds for this modified Delphi process. We, however, acknowledge that surgical health challenges are similar in sub-Saharan Africa, and results from the initial Delphi round that was more representative of sub-Saharan Africa will be useful for several similar African contexts. We also did not have adequate gender balance among the experts, as there are still currently very few female Rwandan surgical providers. Future processes should also consider a deliberate balance of experts by sub-specialty, as sub-specialty training may introduce bias in experts' prioritization preferences.

Conclusions
This consensus process has helped define broad topic areas that are essential for the development of surgical curricula in Rwanda and SSA. This is particularly important as, in an attempt to address the local burden of disease and maximize human resources for health, most graduates from Rwandan medical schools will have 3-to-5-year obligations with the government to practice at rural district hospitals and need the prioritized surgical skills. In this era of globalization, this modified Delphi process has prioritized areas often neglected on global curricula which will also help heighten the global competitiveness of Rwandan medical graduates. These results do not represent a specific curriculum, but a prioritization of surgical teaching for the region. This prioritization raises the bar for competence required of local graduates, which makes them competitive in many parts of the global health market where such surgical skills are not required upon graduation from medical school but are reserved for residency. In addition to representing consensus on what should be taught to medical students with high priority, these results represent an attempt at defining the surgical scope of practice for the non-specialist general practitioner (medical school graduate) in context, which will be updated every five years. The prioritized results have been adopted by surgical training teams at both the University of Rwanda and the University of Global Health Equity and will be implemented in curriculum mapping exercises, the design of EPAs, competency-based medical education, teaching and learning interventions, and simulation-based learning. The result of this consensus process is also useful for surgical education in similar contexts. Priorities for surgical teaching in the context have been clarified, and these should inform the hierarchy of teaching emphasis and the focus of academic resource-and partnershipbuilding.

Appendices Appendix 1: Drafting of initial surgery topic list and defining of broad categories
Important grey literature and web-based sources from which our surgery topic list was developed include the following (  Screening and topic extraction was done by one researcher at title and abstract levels. All surgical undergraduate topics suggested by the article were included if not already present in our curriculum synthesis from grey literature. Full text review of articles that suggested lists of medical school topics for surgery was done by one investigator (BA) and two research assistants. These lists were extracted from text, tables, or figures and integrated into the topic list.

Appendix 2: Included articles and topics derived from PubMed search
Data from search (PubMed) are shown in Table 4 Medical Student And     Table 7.

) Final Delphi Consensus Result Details
Aggregate prioritization score is the sum of assigned scores *Experts in Agreement is the number of experts that rate the item as 'important' (4)